Distance measuring device, distance measuring method, and non-transitory computer-readable storage medium for storing program

ABSTRACT

A distance measuring device is configured to: obtain a first distance image acquired from a time of flight (TOF) sensor and a polarized image generated by calculating a degree of polarization for each pixel based on a plurality of images acquired from a plurality of cameras that receives linearly polarized light with different polarization directions; and execute a process for calculating reliability according to a difference between a time of measuring a distance and a time of photographing the plurality of images for each pixel of the first distance image, and execute a process for calculating a distance from the TOF sensor to a subject for each pixel using a second distance image calculated by weighting the reliability to a distance of each pixel of the first distance image, and a third distance image calculated by estimating a distance of each pixel based on the polarized image.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based upon and claims the benefit of priority of theprior Japanese Patent Application No. 2017-235049, filed on Dec. 7,2017, the entire contents of which are incorporated herein by reference.

FIELD

The embodiments discussed herein are related to a distance measuringdevice, a distance measuring method, and a non-transitorycomputer-readable storage medium for storing a program.

BACKGROUND

One of the ranging techniques for measuring the distance to the objectis a time of flight (TOF) method. In the TOF method, the distance to theobject is measured based on the round-trip time taken for a light beamfor distance measurement emitted to the object to reach the object, bereflected, and return.

Another ranging technique has also been offered in which a degree ofpolarization is calculated based on an image obtained by a camerareceiving linearly polarized light having different polarizationdirections, and the distance to the object is measured based on thedegree of polarization.

Still another ranging technique has also been offered in which both thedistance image acquired from a distance (depth) sensor, and thepolarized image including the degree of polarization obtained from thecamera as described above are used to measure the distance with highaccuracy, compared with the case where a distance sensor alone is used.

Examples of the related art include Japanese Laid-open PatentPublication No. 2013-044597, U.S. Patent Application Publication No.2016/0261844, Seungkyu Lee, “Time-of-Flight Depth Camera MotionBlurDetection and Deblurring”, IEEE SIGNAL PROCESSING LETTERS, VOL. 21, NO.6, JUNE 2014, and Daisuke Miyazaki and Katsushi Ikeuchi, “A Method toEstimate Surface Shape of Transparent Objects by Using PolarizationRaytracing Method”, Transactions of the Institute of Electronics,Information and Communication Engineers D-II, Vol. J88-DII, No. 8,August 2005, pp. 1432-1439.

SUMMARY

According to an aspect of the embodiments, a distance measuring deviceincludes: a memory configured to store a first distance image acquiredfrom a time of flight (TOF) sensor and a polarized image generated bycalculating a degree of polarization for each pixel based on a pluralityof images acquired from a plurality of cameras that receives linearlypolarized light with different polarization directions; and a processorcoupled to the memory and configured to execute a reliabilityacquisition process that includes calculating reliability according to adifference between a time of measuring a distance and a time ofphotographing the plurality of images for each pixel of the firstdistance image, and execute a distance acquisition process that includescalculating an output value of a distance from the TOF sensor to asubject for each pixel using a second distance image calculated byweighting the reliability to a distance of each pixel of the firstdistance image, and a third distance image calculated by estimating adistance of each pixel based on the polarized image.

The object and advantages of the invention will be realized and attainedby means of the elements and combinations particularly pointed out inthe claims.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory and arenot restrictive of the invention.

BRIEF DESCRIPTION OF DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

FIG. 1 is a diagram illustrating an example of a configuration of and anexample of a process of a distance measuring device according to a firstembodiment;

FIG. 2 is a diagram illustrating an example of a configuration of ashape measuring system according to a second embodiment;

FIG. 3 is a diagram illustrating an example of a hardware configurationof a shape measuring device and a sensor unit;

FIG. 4 is a diagram for explaining a problem with a depth measurement bya TOF sensor;

FIG. 5 is a block diagram illustrating an example of a configuration ofa processing function of a shape measuring device;

FIG. 6 is a diagram illustrating an example of a data structure of afirst depth image;

FIG. 7 is a diagram for explaining a calculation of reliability;

FIG. 8 is a diagram illustrating an outline of a process of calculatinga polarized image;

FIG. 9 is a diagram illustrating an example of a relationship between adegree of polarization and a zenith angle;

FIG. 10 is a diagram illustrating an outline of a processing procedureof a detailed shape calculation unit;

FIG. 11 is a flowchart (part 1) illustrating an example of a processingprocedure of the shape measuring device; and

FIG. 12 is a flowchart (part 2) illustrating an example of theprocessing procedure of the shape measuring device.

DESCRIPTION OF EMBODIMENTS

A line scan type TOF sensor is often used to measure a distance. Theline scan type TOF sensor emits a light beam for distance measurement asan irradiation pattern extending in a predetermined first direction, andreceives the reflected light of the light beam while sequentially movingthe irradiation pattern in a second direction orthogonal to the firstdirection.

In such a line scan type TOF sensor, the times at which light is emittedto an object by using the irradiation pattern are different depending onthe positions with respect to the first direction. For this reason, whenthe object to be moved is set as a distance measurement target, theshape of the image of the object captured in the TOF sensor may bedistorted. When the shape of such an image is distorted, there is aproblem that the distance measurement accuracy deteriorates.

In one aspect, the techniques disclosed in the embodiments intend toprovide a distance measuring device, a distance measuring method, and adistance measuring program which are capable of measuring a distance toa moving object with high accuracy.

Hereinafter, embodiments discussed herein will be described withreference to the drawings.

First Embodiment

FIG. 1 is a diagram illustrating an example of a configuration of and anexample of a process of the distance measuring device according to thefirst embodiment. A distance measuring device 1 illustrated in FIG. 1measures the distance between a TOF sensor 2 and an object (notillustrated) to be a distance measurement target for each pixel of theimage of the object using the measurement result by the TOF sensor 2 andimages captured by a plurality of cameras. The TOF sensor 2 emits alight beam for distance measurement to a subject, and measures thedistance (depth) to the subject based on the round trip time from theemission of the light to the reception of the reflected light.

The distance measuring device 1 includes a storage unit la and acalculation unit 1 b. The storage unit la is constructed, for example,as a storage area of a storage device (not illustrated) included in thedistance measuring device 1. The calculation unit 1 b is provided, forexample, as a processor (not illustrated) included in the distancemeasuring device 1.

In the storage unit la stores a distance image 11 (first distance image)acquired from the TOF sensor 2, and a polarized image 12 generated basedon a plurality of images acquired from a plurality of cameras. Althoughin the present embodiment, as an example, three cameras 3 a to 3 c areused for generating the polarized image 12, four or more cameras may beused.

The cameras 3 a, 3 b, and 3 c receive linearly polarized light havingdifferent polarization directions, thereby outputting images 4 a, 4 b,and 4 c, respectively. For example, the camera 3 a receives linearlypolarized light of 0°, the camera 3 b receives linearly polarized lightof 45°, and the camera 3 c receives linearly polarized light of 90°. Thepolarized image 12 is generated by calculating the degree ofpolarization for each pixel based on the images 4 a to 4 c.

The cameras 3 a to 3 c capture the images 4 a to 4 c at the samephotographing time. “The same photographing time” means that theexposure period is the same for the entire images 4 a to 4 c. On theother hand, since the TOF sensor 2 has a structure that may not receivethe reflected light of the distance measuring light beam at the sametime with respect to the entire distance image 11, the times ofmeasuring the distance are partially different for the distance image11. For example, when the line scan type is used, the TOF sensor 2 emitsa light beam for distance measurement by using an irradiation patternextending in a predetermined first direction (for example, alongitudinal direction). The TOF sensor 2 receives reflected light ofthe light beam while sequentially moves the irradiation pattern in asecond direction (for example, a lateral direction) orthogonal to thefirst direction. In this case, the reception times of reflected lightare different depending on the positions on the distance image 11 in thesecond direction, the times of measuring the distance are differentdepending on the positions in the second direction.

The calculation unit 1 b calculates reliability 13 corresponding to thedifference between the time of measuring the distance and thephotographing time by the cameras 3 a to 3 c for each pixel of thedistance image 11. In this calculation, for example, the closer the timeof measuring the distance is to the photographing time, the higher valuethe value of the reliability 13 is set to. The calculation unit 1 bcalculates a distance image 14 (second distance image) by weighting thereliability 13 corresponding to each pixel of the distance image 11.

The calculation unit 1 b calculates a distance image 15 (third distanceimage) by estimating the distance for each pixel based on the polarizedimage 12. The calculation unit 1 b calculates a normal image with whichnormal information is associated for each pixel, for example, based onthe polarized image 12, and calculates the distance image 15 based onthis normal image.

The calculation unit 1 b calculates an output value 16 of the distancefrom the TOF sensor 2 to the subject for each pixel using the distanceimage 14 and the distance image 15. As described above, in the distanceimage 11, the times of measuring the distance are different depending onthe positions of the pixels. For this reason, when the object to bemoved is set as a distance measurement target, the shape of the image ofthe object photographed in the distance image 11 may be distorted. Whenthe shape of the image is distorted, the distance measurement accuracydeteriorates.

On the other hand, the time of photographing the images 4 a to 4 c bythe cameras 3 a to 3 c is the same, the exposure period is the same forthe entire images 4 a to 4 c. Therefore, in the distance image 15calculated from the images 4 a to 4 c, distortion of the shape of theimage caused by the movement of the object as described above does notoccur. Therefore, the calculation unit 1 b may accurately calculate theoutput value 16 of the distance by using not only the distance image 11in which distortion of the shape of the image may occur but also thedistance image 15 in which distortion of the shape of the image does notoccur.

When calculating the output value 16 of the distance, the calculationunit 1 b uses the distance image 14 obtained by converting the distanceimage 11 using the reliability 13 instead of using the distance image 11as it is. The reliability 13 is calculated depending on the differencebetween the time of measuring the distance for each pixel of thedistance image 11, and the time of photographing the images 4 a to 4 c.Therefore, when the output value 16 of the distance is calculated, thedistance image 14 converted using the reliability 13 is used, so thatthe extent to which the measured time and the measured distance isreflected in the process of calculating the output value 16 of thedistance is high in the pixel in which the time of measuring thedistance is close to the photographing time and the accuracy of themeasured distance is high. As a result, when the object moves, it ispossible to reduce the influence of the distortion of the shape of theimage caused by the movement on the accuracy of distance measurement,thereby improving the accuracy of distance measurement.

Second Embodiment

FIG. 2 is a diagram illustrating an example of a configuration of ashape measuring system according to the second embodiment. The shapemeasuring system illustrated in FIG. 2 includes a shape measuring device100 and a sensor unit 200.

The shape measuring device 100 acquires an image of a subject from thesensor unit 200, and measures the depth (distance) between the subjectphotographed in the acquired image and the sensor unit 200 for eachpixel on the image. For example, when an object 300 to be measured isphotographed in the image, the shape measuring device 100 measures thedepth for each pixel of the image with respect to the object 300photographed in the image. As a result, the shape measuring device 100may measure the three-dimensional shape of the object 300.

The sensor unit 200 includes a TOF sensor 210 and a camera unit 220. TheTOF sensor 210 calculates the depth of the subject using the TOF method.For example, the TOF sensor 210 calculates the depth of the subjectbased on the round trip time of laser light taken for laser lightemitted from the TOF sensor 210 to reach the subject, be reflected, andreturn. Hereinafter, the depth calculated by the TOF sensor 210 isreferred to as a first depth. The TOF sensor 210 outputs to the shapemeasuring device 100 the first depth image obtained by plotting thedepth for each pixel on the image.

On the other hand, the camera unit 220 simultaneously captures aplurality of camera images obtained by receiving components of linearlypolarized light having different polarization directions out of thelight from the subject, and outputs these camera images to the shapemeasuring device 100. The shape measuring device 100 may acquire aluminance image as each camera image. For this purpose, the camera unit220 may be configured to capture a luminance image, alternatively, thecamera unit 220 may capture a color image, and the shape measuringdevice 100 may convert the color image into a luminance image.

The shape measuring device 100 acquires the first depth image based onthe first depth output from the TOF sensor 210. In addition, the shapemeasuring device 100 calculates the polarized image based on the cameraimage output from the camera unit 220, and then calculates the depth ofthe subject based on the calculated polarized image. Hereinafter, thedepth obtained from the polarized image based on the camera image isreferred to as a second depth. As will be described later, the shapemeasuring device 100 uses both the first depth and the second depth,thereby calculating the depth with high accuracy with reduced influenceof the distortion of the shape of the object 300 appearing in themeasurement result of the first depth.

FIG. 3 is a diagram illustrating an example of a hardware configurationof the shape measuring device and the sensor unit. First, the shapemeasuring device 100 is provided, for example, as a computer asillustrated in FIG. 3. The shape measuring device 100 illustrated inFIG. 3 includes a processor 101, a random access memory (RAM) 102, ahard disk drive (HDD) 103, a graphics processing device 104, an inputinterface 105, a reading device 106, a network interface 107 and acommunication interface 108.

The processor 101 controls the entire shape measuring device 100. Theprocessor 101, for example, a central processing unit (CPU), a microprocessing unit (MPU), a digital signal processor (DSP), an applicationspecific integrated circuit (ASIC) or a programmable logic device (PLD).The processor 101 may be a combination of two or more of the CPU, theMPU, the DSP, the ASIC, and the PLD.

The RAM 102 is used as a main storage device of the shape measuringdevice 100. The RAM 102 temporarily stores at least part of an operatingsystem (OS) program and an application program both of which areexecuted by the processor 101. The RAM 102 stores various data which isused for processing by the processor 101.

The HDD 103 is used as an auxiliary storage device of the shapemeasuring device 100. The HDD 103 stores the OS program, the applicationprogram, and various data. As the auxiliary storage device, another typeof nonvolatile storage device such as a solid state drive (SSD) may alsobe used.

A display device 104 a is connected to the graphics processing device104. The graphics processing device 104 causes the display device 104 adisplays an image according to a command from the processor 101.Examples of the display device include a liquid crystal display and anorganic electroluminescence (EL) display.

An input device 105 a is connected to the input interface 105. The inputinterface 105 transmits a signal output from the input device 105 a tothe processor 101. Examples of the input device 105 a include a keyboardand a pointing device. Examples of the pointing device include a mouse,a touch panel, a tablet, a touch pad, and a track ball.

A portable recording medium 106 a is removably attached to the readingdevice 106. The reading device 106 reads the data recorded on theportable recording medium 106 a and transmits the read data to theprocessor 101. Examples of the portable recording medium 106 a includean optical disk, a magneto-optical disk, and a semiconductor memory.

The network interface 107 transmits and receives data to and from otherdevices via a network 107 a. The communication interface 108 transmitsand receives data to and from the sensor unit 200.

The hardware configuration as described above may implement theprocessing function of the shape measuring device 100. The TOF sensor210 of the sensor unit 200 includes an infrared light emitting unit 211and an infrared light receiving unit 212. The infrared light emittingunit 211 emits an infrared laser to the subject. The infrared lightreceiving unit 212 receives the reflected light of the infrared laseremitted from the infrared light emitting unit 211. The TOF sensor 210calculates the depth of the subject by measuring the round trip timetaken for the infrared laser emitted from the infrared light emittingunit 211 to reach the subject, be reflected, and return, and multiplyingthe measured round trip time by the light speed. The TOF sensor 210outputs to the shape measuring device 100 the first depth image obtainedby plotting the depth for each pixel on the image.

The camera unit 220 of the sensor unit includes cameras 221 to 223 andpolarizing filters 221 a to 223 a. Each of the cameras 221 to 223, forexample, photographs a red/blue/green (RGB) color image. The polarizingfilters 221 a to 223 a are linearly polarizing plates that transmitlinearly polarized light having a specific polarization direction amongthe incident light.

A polarizing filter 221 a is disposed with a rotation angle of 0° aroundthe optical axis of the camera 221 on the imaging surface of the camera221, and the camera 221 captures an image via the polarizing filter 221a. The polarizing filter 222 a is disposed with a rotation angle of 45°around the optical axis of the camera 222 on the imaging surface of thecamera 222, and the camera 222 captures an image via the polarizingfilter 222 a. The polarizing filter 223 a is disposed with a rotationangle of 90° around the optical axis of the camera 223 on the imagingsurface of the camera 223, and the camera 223 captures an image via thepolarizing filter 223 a. As a result, the camera 221 outputs a cameraimage obtained by receiving linearly polarized light of 0°, the camera222 outputs a camera image obtained by receiving linearly polarizedlight of 45°, an the camera 223 outputs a camera image obtained byreceiving linearly polarized light of 90°.

Although not illustrated, each of the cameras 221 to 223 has a globalshutter. The cameras 221 to 223 open and close each global shuttersynchronously, so that it is possible to output a camera image shot atthe same photographing time (for example, exposed in the same exposureperiod).

Next, FIG. 4 is a diagram for explaining the problem with a depthmeasurement by the TOF sensor. The infrared light emitting unit 211 ofthe TOF sensor 210 emits the infrared laser with a linear irradiationpattern, and gradually shifts the irradiation pattern, thereby emittingthe infrared laser over the entire depth measurement range. In theexample illustrated on the left side of FIG. 4, the linear irradiationpattern is indicated by a broken line. In this example, the irradiationpattern has a linear shape extending in the vertical direction, andirradiation is performed such that the irradiation pattern is graduallyshifted from right to left. Hereinafter, the shift direction of theirradiation pattern is referred to as “a scan direction”.

Since the infrared laser emits light in this way, in the first depthimage output from the TOF sensor 210, the scan times (reception times ofreflected light of the infrared laser) are different depending on thepositions with respect to the scan direction. This means that, forexample, when measuring the three-dimensional shape of the moving object300, the measurement result is inaccurate.

FIG. 4 illustrates a case where the scanning direction is from right toleft, and the object 300 moves from the top to the bottom. In this case,as illustrated on the right side of FIG. 4, an image 301 of the object300 photographed in the first depth image is shaped such that its leftside is shifted downward. In this way, when the object 300 moves, theshape of the image 301 of the object 300 may be distorted. Such adistorted shape of the image 301 has a problem that the measured firstdepth is inaccurate.

An increase in the scanning speed of the TOF sensor 210 may reduce theoccurrence of distortion as described above, and improve the measurementaccuracy of the first depth. However, since the scan speed and thenumber of scans in one image (the irradiation number of the irradiationpattern with respect to the scan direction) are in a trade-offrelationship, the number of scans will decrease when the scan speed isincreased. As a result, there is a problem that the resolution of thefirst depth image is lower, whereby the obtained three-dimensional shapeis rough.

In order to cope with such a problem, the shape measuring device 100 ofthe present embodiment calculates the depth of the subject using thesecond depth image in addition to the first depth image. The seconddepth image is obtained from the calculated polarized image based on thecamera image acquired from the camera unit 220. As described above, thecamera unit 220 photographs at the same time a plurality of cameraimages obtained by receiving linearly polarized light having differentpolarization directions by using the global shutter. Therefore, in thepolarized image obtained from such a camera image, the shape of theimage 301 of the object 300 is not distorted. Utilizing this property,the shape measuring device 100 calculates the depth with high accuracyin which the influence of the distortion of the shape of the imageappearing in the first depth image is reduced by using both the firstdepth image and the second depth image.

FIG. 5 is a block diagram illustrating an example of a configuration ofa processing function of the shape measuring device. The shape measuringdevice 100 includes a storage unit 110, a first depth acquisition unit121, a camera image acquisition unit 122, a reliability calculation unit123, a polarized image calculation unit 124, a normal calculation unit125 and a detailed shape calculation unit 126.

The storage unit 110 is constructed as a storage area of a storagedevice, such as the RAM 102 and the HDD 103, which is included in theshape measuring device 100. The storage unit 110 stores a first depthimage 111, camera images 112 a to 112 c, a reliability map 113, apolarized image 114, a normal image 115 and a second depth image 116.

The processing of the first depth acquisition unit 121, the camera imageacquisition unit 122, the reliability calculation unit 123, thepolarized image calculation unit 124, the normal calculation unit 125and the detailed shape calculation unit 126 are, for example,implemented by the processor 101 executing a predetermined program.

The first depth acquisition unit 121 acquires the first depth image 111from the TOF sensor 210, and stores the acquired first depth image 111in the storage unit 110. The first depth image 111 has a structure inwhich the measured values of the depth are plotted for each pixel on theimage. The acquisition time of the first depth image 111 is added to thedata of the first depth image 111.

The camera image acquisition unit 122 acquires camera images 112 a, 112b, and 112 c photographed by the cameras 221, 222, and 223 of the cameraunit 220, respectively, and stores the acquired camera images 112 a, 112b, and 112 c in the storage unit 110. A photographing time is added toeach data of the camera images 112 a to 112 c. Since the camera images112 a to 112 c are photographed at the same time, the photographing timeadded is also the same.

The reliability calculation unit 123 calculates the reliability for eachpixel on the first depth image 111 based on the acquisition time of thefirst depth image 111, and the time of photographing the camera images112 a to 112 c. With respect to the reliability, the closer the scantime is to the time of photographing the camera images 112 a to 112 cwith respect to the aforementioned scan direction in the first depthimage 111, the larger the assigned value is. The scan time is convertedfrom the acquisition time of the first depth image 111. The reliabilitycalculation unit 123 generates the reliability map 113 in which thereliability is plotted for each pixel on the first depth image 111, andstores the generated reliability map 113 in the storage unit 110.

The polarized image calculation unit 124 calculates the polarized image114 based on the camera images 112 a to 112 c, and stores the calculatedpolarized image 114 in the storage unit 110. The polarized image 114 hasa structure in which the degree of polarization is plotted for eachpixel on the image. The degree of polarization is an index indicatinghow much light is polarized, is a value of 0 or more and 1 or less.

The normal calculation unit 125 calculates the normal information foreach pixel on the polarized image 114 based on the polarized image 114.The normal calculation unit 125 generates the normal image 115 in whichnormal information is plotted for each pixel on the image, and storesthe generated normal image 115 in the storage unit 110.

The detailed shape calculation unit 126 estimates the second depth foreach pixel of the normal image 115 based on the normal informationincluded in the normal image 115. The detailed shape calculation unit126 stores in the storage unit 110 the second depth image 116 in whichthe second depth is plotted for each pixel.

The detailed shape calculation unit 126 multiplies the depth of eachpixel of the first depth image 111 by the corresponding reliabilityincluded in the reliability map 113, thereby converting the first depthimage 111. The depth estimated from the normal image 115 has an unknownscale. The detailed shape calculation unit 126 performs the process ofminimizing the squared error between a first depth image converted usingreliability and the second depth image 116, thereby estimating the scaleof the second depth image 116. The detailed shape calculation unit 126outputs A second depth image to which the estimated scale is applied asa depth image in which the distortion of the shape of the image iscorrected.

<Acquisition of First Depth Image and Camera Image>

Hereinafter, the processing of the shape measuring device 100 will bedescribed in more detail. The first depth acquisition unit 121 acquiresthe first depth image 111 from the TOF sensor 210, and stores theacquired first depth image 111 in the storage unit 110. The camera imageacquisition unit 122 acquires the camera images 112 a, 112 b, and 112 cphotographed by the cameras 221, 222, and 223 of the camera unit 220,respectively, and stores the acquired camera images 112 a, 112 b, and112 c in the storage unit 110.

The camera images 112 a to 112 c are stored in the storage unit 110 asluminance images. When color images are output from the cameras 221,222, and 223, the camera image acquisition unit 122 converts the colorimages output from the cameras 221, 222, and 223 into luminance images,and stores in a storage unit 110 the converted color images as cameraimages 112 a, 112 b, and 112 c. It is assumed that the camera images 112a to 112 c are stored in the storage unit 110 as images having aresolution same as the first depth image 111.

As described above, the cameras 221, 222, and 223 photograph the cameraimages 112 a, 112 b, and 112 c, respectively, and output them at thesame time by using the global shutter. When the data of the acquiredcamera images 112 a to 112 c in the storage unit 110 is stored, thecamera image acquisition unit 122 adds the simultaneous photographingtime to each data.

On the other hand, when storing the data of the acquired first depthimage 111 in the storage unit 110, the first depth acquisition unit 121adds the acquisition time of the first depth image 111 to the data. Thisacquisition time may be, for example, the time at which the first depthacquisition unit 121 instructs the TOF sensor 210 to output the firstdepth image 111, the time at which the first depth acquisition unit 121completes the reception of the first depth image 111, or the like.

The time of photographing the camera images 112 a to 112 c and theacquisition time of the first depth image 111 are referred to when thereliability calculation unit 123 calculates the reliability. Asdescribed with reference to FIG. 4, in the first depth image 111, thescan times differs depending on the positions in the scan direction. Thereliability calculation unit 123 may, from the acquisition time of thefirst depth image 111 as described above, convert the scan time for eachposition with respect to the scan direction.

The acquisition of the first depth image 111 and the acquisition of thecamera images 112 a to 112 c are performed synchronously. For example,the first depth image 111 and the camera images 112 a to 112 c areacquired so that the time of photographing the camera images 112 a to112 c is included in the scan period of the first depth image 111 (theperiod from the scan time at the position at one end with respect to thescan direction to the scan time at the position of the other end).

FIG. 6 is a diagram illustrating an example of the data structure of thefirst depth image. The data of the first depth image 111 is stored inthe storage unit 110 as the data of the structure associated with theposition in the X-axis direction (horizontal direction) the position inthe Y-axis direction (vertical direction), and the position in the Zdirection (depth direction) for each pixel. Among them, the position inthe Z direction indicates the depth (distance). The data of the firstdepth image 111 is stored, for example, as a data table 111 a asillustrated in FIG. 6. In the data table 111 a illustrated in FIG. 6,each position in the X direction, the Y direction, and the Z directionis registered in association with the pixel number of each pixel.

<Calculation of Reliability>

Next, a process of calculating the reliability by the reliabilitycalculation unit 123 will be described. FIG. 7 is a diagram forexplaining calculation of the reliability. The reliability calculationunit 123 calculates the reliability for each pixel on the first depthimage 111 based on the acquisition time of the first depth image 111 andthe time of photographing the camera images 112 a to 112 c.

For example, the reliability calculation unit 123 calculates the scantime for each position with respect to the scan direction in the firstdepth image 111 based on the acquisition time of the first depth image111. In the example of the first depth image 111 illustrated in FIG. 7,the scan direction is from right to left. In this case, different scantimes are calculated for respective pixel columns having the samehorizontal coordinate (for example, pixel columns arranged in parallelin the vertical direction).

Next, the reliability calculation unit 123 calculates the reliabilitycorresponding to the difference between the scan time and the time ofphotographing the camera images 112 a to 112 c for each pixel of thefirst depth image 111. The reliability is calculated as a value of 0 ormore and 1 or less. The closer the scanning time is to the photographingtime, the higher the value of reliability is. When the scan timecoincides with the photographing time, the reliability is set to 1. Thereason for this is that it is probable that the closer the scan time isto the photographing time, the higher degree of coincidence the positioninformation of the image obtained at the scan time and the positioninformation of the image in the second depth image 116 calculated basedon the camera images 112 a to 112 c has. Such reliability is used inwhich the process of minimizing the square error between the first depthimage 111 and the second depth image 116 by the detailed shapecalculation unit 126 is performed, whereby weighting according to thereliability is performed for the square error for each pixel. As aresult, the accuracy of the process of minimizing the squared error maybe improved.

A graph 113 a in FIG. 7 illustrates an example of the relationshipbetween the respective coordinates in the X-axis direction and theY-axis direction, and the reliability in the first depth image 111. Inthis example, the time of photographing the camera images 112 a to 112 cis t. In this case, reliability is calculated to be 1 for a pixel columnwhose scan time is t among the pixel columns having the same horizontalcoordinate in the first depth image 111. The reliability is calculatedto be a value which is smaller than 1 with respect to a pixel columnwhose scan time is (t−m), and a pixel column whose scan time is (t+m).In the example of FIG. 7, the reliability is calculated to be the samevalue with respect to these respective pixel columns.

Information indicating the relationship between the differential valuebetween the scan time and the photographing time, and the reliability isprepared in advance, for example, as an expression or a data table, andis stored in the storage unit 110. The reliability calculation unit 123calculates the reliability corresponding to the differential valuebetween the scan time and the photographing time based on suchinformation.

Through the above procedure, the reliability calculation unit 123calculates the reliability for each pixel of the first depth image 111.The reliability calculation unit 123 generates the reliability map 113in which the reliability is plotted for each pixel of the first depthimage 111, and stores the generated reliability map 113 in the storageunit 110.

<Calculation of Polarized Image>

Next, the process of calculating the polarized image 114 by thepolarized image calculation unit 124 will be described. FIG. 8 is adiagram illustrating an outline of a process of calculating a polarizedimage. The polarized image calculation unit 124 performs a viewpointconversion process for matching the viewpoint to the viewpoint of thefirst depth image 111 for each of the camera images 112 a to 112 c. Thefollowing preparations are made in order to perform this viewpointconversion process.

External parameters of the TOF sensor 210 and the cameras 221 to 223 areset based on the positions and orientations of the TOF sensor 210 andthe cameras 221 to 223. Internal parameters of the cameras 221 to 223are set. These parameters are obtained by calibration. A homographymatrix for converting the viewpoints of the cameras 221 to 223 to theviewpoint of the TOF sensor 210 is calculated based on these parameters.

The polarized image calculation unit 124 calculates camera images 112 a1, 112 b 1, and 112 c 1 by converting the camera images 112 a, 112 b,and 112 c using the homography matrix. The camera images 112 a 1, 112 b1, and 112 c 1 are images obtained when the camera images 112 a, 112 b,and 112 c are photographed in the same direction from the same positionas the TOF sensor 210.

Next, the polarized image calculation unit 124 calculates the polarizedimage 114 using the camera images 112 a 1 to 112 c 1 obtained by theconversion. In this calculation process, the following processing isperformed for each pixel of the camera images 112 a 1 to 112 c 1.

The polarized image calculation unit 124 approximates the luminancevalue of the m-th pixel in the camera images 112 a 1 to 112 c 1 by thecosine curve expressed by the following expression (1).

y _(m) =a _(m) cos(Θ+b _(m))+c _(m)  (1)

The polarized image calculation unit 124 obtains the maximum value andthe minimum value in the obtained cosine curve, and calculates thedegree of polarization of the m-th pixel by the expression of (maximumvalue/minimum value)/(maximum value+minimum value). As a result, thedegree of polarization is calculated as a value of 0 or more and 1 orless.

The polarized image calculation unit 124 calculates the degree ofpolarization for each pixel according to the above processing procedure.The polarized image calculation unit 124 generates the polarized image114 in which the degree of polarization is plotted for each pixel, andstores the generated polarized image 114 in the storage unit 110.

The degree of polarization may be calculated from a camera imageobtained by using three or more polarizing filters with differentrotation angles. In the present embodiment, as an example, although thedegree of polarization is calculated based on the three camera images112 a to 112 c obtained by using the three polarizing filters 221 a to223 a, the degree of polarization may be calculated based on cameraimages obtained by using four or more polarizing filters.

<Calculation of Normal Information>

Next, the process of calculating the normal information by the normalcalculation unit 125 will be described. The normal calculation unit 125calculates normal information for each pixel on the polarized image 114based on the polarized image 114. The zenith angle θ and the azimuthangle φ are calculated as normal information.

FIG. 9 is a diagram illustrating an example of the relationship betweenthe degree of polarization and the zenith angle. Assuming that theobject 300 is a diffusely reflecting object, the zenith angle θ isuniquely determined from the degree of polarization. The relationshipbetween the degree of polarization and the zenith angle θ is, forexample, represented by a graph 125 a illustrated in FIG. 9. The normalcalculation unit 125 calculates the normal calculates the zenith angle θfor each pixel from the degree of polarization based on thisrelationship.

On the other hand, the azimuth angle φ is calculated by performing thefollowing process for each pixel. The normal calculation unit 125calculates the angle θ_(max) at which the luminance value is maximizedin the cosine curve approximated in the process of calculating thepolarized image 114. This angle θ_(max) represents the polarizationdirection in which the amount of received linearly polarized light ismaximized. This angle θ_(max) is calculated as b_(m) in the equation(1).

The azimuth angle φ is either the calculated angle θ_(max) or the angleobtained by adding 180° to this angle θ_(max). The normal calculationunit 125 specifies one of θ_(max) and (θ_(max)+180) as an azimuth angleφ by the following procedure based on the normal information obtainedfrom the first depth image 111.

Let z=f (x, y) be the function representing the depth z at thecoordinates (x, y). Let the gradient of the first depth at thecoordinates (x, y) of the first depth image 111 be (p, q, −1). Asillustrated in the following expressions (2-1) and (2-2), p and q areobtained by partially differentiating the first depth at the coordinates(x, y) of the first depth image 111. There is a relationship illustratedin the following expressions (3-1) to (3-3) between the gradient (p, q,−1) and the normal n=(n_(x), n_(y), n_(z)) at the coordinates (x, y) ofthe first depth image 111.

$\begin{matrix}{p = \frac{\delta \; f}{\delta \; x}} & \left( {2\text{-}1} \right) \\{q = \frac{\delta \; f}{\delta \; y}} & \left( {2\text{-}2} \right) \\{n_{x} = \frac{p}{\sqrt{p^{2} + q^{2} + 1}}} & \left( {3\text{-}1} \right) \\{n_{y} = \frac{q}{\sqrt{p^{2} + q^{2} + 1}}} & \left( {3\text{-}2} \right) \\{n_{z} = \frac{1}{\sqrt{p^{2} + q^{2} + 1}}} & \left( {3\text{-}3} \right)\end{matrix}$

The normal calculation unit 125 calculates the azimuth angle φ′ at thecoordinates (x, y) of the first depth image 111 using the aboveequations (2-1), (2-2), (3-1) to (3-3). The normal calculation unit 125specifies the angle closer to the calculated azimuth angle φ′ among theθ_(max) and the (θ_(max)+180) described above as the azimuth angle φbased on the polarized image 114.

In the process of calculating the azimuth angle φ, the angle of thepolarizing filter corresponding to the camera image having the largestluminance value among the three camera images 112 a 1 to 112 c 1 may beobtained as the angle θ_(max) without using the cosine curveapproximated as described above.

The normal calculation unit 125 calculates the zenith angle θ and theazimuth angle φ for each pixel according to the above processingprocedure. The normal calculation unit 125 generates the normal image115 by plotting the zenith angle θ and the azimuth angle φ with respectto each pixel on the image, and stores the generated normal image 115 inthe storage unit 110.

<Calculation of Detailed Shape>

Next, the processing of the detailed shape calculation unit 126 will bedescribed. FIG. 10 is a diagram illustrating the outline of theprocessing procedure of the detailed shape calculation unit.

The detailed shape calculation unit 126 multiplies the first depth ofeach pixel of the first depth image 111 by the reliability of thecorresponding pixel included in the reliability map 113. By suchmultiplication, the first depth image 111 is converted into a convertedfirst depth image 117 illustrated in FIG. 10.

The detailed shape calculation unit 126 estimates the depth (seconddepth) for each pixel of the normal image 115 based on the normalinformation included in the normal image 115, thereby calculating thesecond depth image 116. The second depth estimated at this time is anvalue with a indefinite scale. Therefore, the detailed shape calculationunit 126 performs the process of minimizing the squared error for eachpixel between the depth (converted first depth) of the converted firstdepth image 117 obtained by the above conversion and the second depth ofthe second depth image 116. In this process, the scale of the seconddepth is estimated by the least square method based on the magnitude ofthe converted first depth based on the measurement result of the TOFsensor 210.

Hereinafter, the second depth estimation and the process of minimizingthe squared error between the converted first depth and the second depthwill be described in detail. The detailed shape calculation unit 126estimates the second depth of each pixel of the normal image 115 in thefollowing procedure, thereby generating the second depth image 116. Thenormal line n′=(n′_(x), n′_(y), n′_(z)) at the coordinates (x, y) of thenormal image 115 is expressed by the following equations (4-1) to (4-3)using the zenithal angle θ and the azimuth angle φ.

n′ _(x)=cos φ sin θ  (4-1)

n′ _(y)=sin φ cos θ  (4-2)

n′ _(z)=cos θ  (4-3)

The normal n′=(n′_(x), n′_(y), n′_(z)) is expressed by the followingexpressions (5-1) to (5-3) where the gradient of the second depth at thecoordinates (x, y) of the normal image 115 is (p′, q′, −1).

$\begin{matrix}{n_{x}^{\prime} = \frac{p^{\prime}}{\sqrt{p^{\prime^{2}} + q^{\prime^{2}} + 1}}} & \left( {5\text{-}1} \right) \\{n_{y}^{\prime} = \frac{q^{\prime}}{\sqrt{p^{\prime^{2}} + q^{\prime^{2}} + 1}}} & \left( {5\text{-}2} \right) \\{n_{z}^{\prime} = \frac{1}{\sqrt{p^{\prime^{2}} + q^{\prime^{2}} + 1}}} & \left( {5\text{-}3} \right)\end{matrix}$

The detailed shape calculation unit 126 multiplies the normal n′ ofequations (5-1) to (5-3) by n′_(z), and calculates P′ and q′ which areparameters indicating the gradient of the second depth based on themultiplication result and the expressions (4-1) to (4-3). The detailedshape calculation unit 126 calculates the second depth Dp using thefollowing expression (6).

z=Dp=(∫p′dx+∫p′dy)/2  (6)

In the above estimation of the second depth Dp, the second depth Dp isestimated by utilizing the relationship in which the normal line may beobtained by partially differentiating the gradient of the depth asillustrated by equations (2-1) and (2-2) and by performing lineintegration based on the normal information obtained from the polarizedimage 114. The detailed shape calculation unit 126 generates the seconddepth image 116 by plotting the second depth Dp estimated usingexpression (6) for each pixel, and stores the generated second depthimage 116 in the storage unit 110.

The second depth image 116 is converted from the camera images 112 a to112 c. Therefore, unlike the first depth image 111, in the second depthimage 116, distortion of the shape of the image caused by the movementof the object 300 does not occur. On the other hand, however, the scaleof the second depth Dp estimated from the above equation (6) isindefinite, whereby its absolute value is not determined correctly.

For this reason, the detailed shape calculation unit 126 estimates thescale of the second depth image 116 by performing the process ofminimizing the squared error between the first depth of the first depthimage 111 and the second depth Dp of the second depth image 116. At thistime, the detailed shape calculation unit 126 does not directly use thefirst depth image 111. As illustrated in FIG. 10, estimation isperformed by using the converted first depth image 117 to which thereliability based on the difference between the photographing time andthe scan time is applied. As a result, weighting according to thereliability is performed in the process of minimizing the squared error,and the processing accuracy is improved.

The detailed shape calculation unit 126 performs the following processusing, for example, the least squares method. The converted first depthin the m-th pixel of the converted first depth image 117 is Dc_(m), andthe second depth in the m-th pixel of the second depth image 116 isDp_(m). The weight used for the process of minimizing the squared erroris w, and the variable for determining the scale is β. The detailedshape calculation unit 126 calculates the variable β at which theresidual sum of squares RSS expressed by the following equation (7) isminimized.

$\begin{matrix}{{RSS} = {\sum\limits_{j = 0}^{m}\; \left\{ {{wDc}_{m} - {\beta \; {Dp}_{m}}} \right\}^{2}}} & (7)\end{matrix}$

The detailed shape calculation unit 126 outputs a second depth image towhich the estimated scale is applied, for example, βDp_(m) calculatedfor each pixel, as a final depth image in which the influence of thedistortion of the image shape is reduced. The depth of the final depthimage is calculated by applying the variable β to the second depthestimated from the camera images 112 a to 112 c in which distortion ofthe shape of the image does not occur even when the object 300 moves.Therefore, it is possible to calculate the depth with high accuracywhere the influence of the distortion of the shape of the image isreduced. As a result, even when the object 300 moves, thethree-dimensional shape of the object 300 may be measured with highaccuracy.

<Flowchart>

Next, the processing procedure of the shape measuring device 100 will bedescribed with reference to a flowchart. FIGS. 11 and 12 are flowchartsillustrating an example of the processing procedure of the shapemeasuring device.

[Step S11] The first depth acquisition unit 121 acquires the first depthimage 111 from the TOF sensor 210, and stores the acquired first depthimage 111 in the storage unit 110. The camera image acquisition unit 122acquires the camera images 112 a, 112 b, and 112 c photographed by thecameras 221, 222, and 223 of the camera unit 220, respectively, andstores the acquired camera images 112 a, 112 b, and 112 c in the storageunit 110.

[Step S12] The polarized image calculation unit 124 performs theviewpoint conversion process for matching the viewpoint to the viewpointof the first depth image 111 for each of the camera images 112 a to 112c. For example, the polarized image calculation unit 124 calculatescamera images 112 a 1, 112 b 1, and 112 c 1 by converting the cameraimages 112 a, 112 b, and 112 c using the homography matrix for theviewpoint conversion.

[Step S13] The polarized image calculation unit 124 calculates thepolarized image 114 from the camera images 112 a 1 to 112 c 1 convertedin step S12. For example, the polarized image calculation unit 124approximates the luminance value of the m-th pixel in the camera images112 a 1 to 112 c 1 by the cosine curve represented by theabove-described expression (1). The polarized image calculation unit 124obtains the maximum value and the minimum value in the obtained cosinecurve, and calculates the degree of polarization of the m-th pixel bythe expression of (maximum value/minimum value)/(maximum value+minimumvalue). The polarized image calculation unit 124 generates the polarizedimage 114 in which the degree of polarization is plotted for each pixel,and stores the generated polarized image 114 in the storage unit 110.

[Step S14] The reliability calculation unit 123 performs the process ofsteps S15 and S16 for each pixel column of the first depth image 111.The pixel column is a group of pixels arranged in parallel in adirection orthogonal to the scanning direction of the TOF sensor 210 inthe first depth image 111. For example, when the scan direction ishorizontal as illustrated in FIG. 7, the pixel columns are a group ofpixels arranged in parallel in the vertical direction.

[Step S15] The reliability calculation unit 123 calculates the scan timeof the first depth for the pixel column to be processed based on theacquisition time of the first depth image 111. The reliabilitycalculation unit 123 acquires the time of photographing the cameraimages 112 a to 112 c, and compares the calculated scan time with theacquired photographing time.

[Step S16] The reliability calculation unit 123 determines thereliability according to the difference between the scan time and thephotographing time. The reliability is determined based on apredetermined rule such that the reliability is a value of 0 or more and1 or less, and the closer the scanning time is to the photographingtime, the higher the value of reliability is.

[Step S17] After the process of steps S15 and S16 is performed for allthe pixel columns of the first depth image 111, the reliabilitycalculation unit 123 generates the reliability map 113 in which thereliability is plotted for each pixel of the first depth image 111, andstores the generated reliability map 113 in the storage unit 110. Thereliability calculation unit 123 advances the process to step S21 inFIG. 12.

[Step S21] The normal calculation unit 125 calculates the zenith angle θand the azimuth angle φ as normal information for each pixel of thepolarized image 114 based on the polarized image 114. For example, thenormal calculation unit 125 calculates the zenith angle θ correspondingto the degree of polarization of each pixel of the polarized image 114,for example, based on a predetermined relationship between the degree ofpolarization and the zenith angle θ as illustrated in the graph 125 a inFIG. 9. The azimuth angle φ is calculated as follows.

The normal calculation unit 125 calculates the angle θ_(max) at whichthe luminance value is maximized in the cosine curve approximated in theprocess of calculating the polarized image 114. The normal calculationunit 125 calculates the azimuth angle φ′ at the coordinates (x, y) ofthe first depth image 111 using the above equations (2-1), (2-2), and(3-1) to (3-3). The normal calculation unit 125 specifies the anglecloser to the calculated azimuth angle φ′ among the θ_(max) and the(θ_(max)+180) as the azimuth angle φ based on the polarized image 114.

The normal calculation unit 125 generates a normal image 115 by plottingthe zenith angle θ and the azimuth angle φ with respect to each pixel ofthe polarized image 114, and stores the generated normal image 115 inthe storage unit 110.

[Step S22] The detailed shape calculation unit 126 estimates the seconddepth of each pixel of the normal image 115 based on the normal image115. For example, the detailed shape calculation unit 126 multiplies thenormal n′ of the above-described expressions (5-1) to (5-3) by n′_(z)and calculates p′ and q′ which are parameters indicating the gradient ofthe second depth based on the multiplication result and theabove-described equations (4-1) to (4-3). The detailed shape calculationunit 126 calculates the second depth Dp using the following expression(6). The detailed shape calculation unit 126 generates the second depthimage 116 by plotting the estimated second depth Dp for each pixel, andstores the generated second depth image 116 in the storage unit 110.

[Step S23] The detailed shape calculation unit 126 multiplies the firstdepth of each pixel of the first depth image 111 by the reliability ofthe corresponding pixel included in the reliability map 113. As aresult, the first depth image 111 is converted into the converted firstdepth image 117.

The processes of steps S12 to S17, and S21 to S24 may be changed withinthe scope in which the process order of steps S12, S13, S21, and S22,and the process order of S14 to S17 and S23 are maintained, and theprocess order in which step S22 is performed after step S13 ismaintained.

[Step S24] The detailed shape calculation unit 126 performs a process ofminimizing the squared error between the converted first depth of theconverted first depth image 117 generated at step S23 and the seconddepth Dp of the second depth image 116 generated at step S22. As aresult, the detailed shape calculation unit 126 estimates the scale ofthe second depth image 116. For example, the detailed shape calculationunit 126 calculates the variable β such that the residual sum of squaresRSS expressed by the above-mentioned (7) is minimized. The detailedshape calculation unit 126 outputs a second depth image to which theestimated scale is applied, for example, βDp_(m) calculated for eachpixel, as a final depth image in which the influence of the distortionof the image shape is reduced.

The processing functions of the devices (the distance measuring device 1and the shape measuring device 100) illustrated in each of the aboveembodiments may be implemented by a computer. In this case, theabove-described processing function is implemented in the computer byproviding a program describing processing contents of the function whicheach device has and executing the program with the computer. The programdescribing the processing contents may be recorded in a computerreadable recording medium. The computer readable recording mediumincludes a magnetic storage device, an optical disk, a magneto-opticalrecording medium, a semiconductor memory, and so forth. The magneticmemory device includes a hard disk device (HDD), a flexible disk (FD), amagnetic tape, and the like. The optical disk includes a digitalversatile disc (DVD), a DVD-RAM, a compact disc-read only memory(CD-ROM), a CD-recordable (R)/rewritable (RW), and the like.

When the program is distributed, for example, a portable recordingmedium, such as a DVD and a CD-ROM, in which the program is recorded maybe sold. The program may be stored in a storage device of a servercomputer and the program may be transmitted from the server computer toanother computer via a network.

The computer executing the program stores the program recorded in theportable recording medium or the program transmitted from the servercomputer in a storage device thereof. The computer reads the programfrom the storage device thereof and executes processing according to theprogram. The computer may directly read the program from the portablerecording medium and execute processing according to the program. Thecomputer may execute the processing according to the sequentiallyreceived programs whenever the program is transmitted from the servercomputer coupled via the network.

All examples and conditional language provided herein are intended forthe pedagogical purposes of aiding the reader in understanding theinvention and the concepts contributed by the inventor to further theart, and are not to be construed as limitations to such specificallyrecited examples and conditions, nor does the organization of suchexamples in the specification relate to a showing of the superiority andinferiority of the invention. Although one or more embodiments of thepresent invention have been described in detail, it should be understoodthat the various changes, substitutions, and alterations could be madehereto without departing from the spirit and scope of the invention.

What is claimed is:
 1. A distance measuring device comprising: a memoryconfigured to store a first distance image acquired from a time offlight (TOF) sensor and a polarized image generated by calculating adegree of polarization for each pixel based on a plurality of imagesacquired from a plurality of cameras that receives linearly polarizedlight with different polarization directions; and a processor coupled tothe memory and configured to execute a reliability acquisition processthat includes calculating reliability according to a difference betweena time of measuring a distance and a time of photographing the pluralityof images for each pixel of the first distance image, and execute adistance acquisition process that includes calculating an output valueof a distance from the TOF sensor to a subject for each pixel by using asecond distance image calculated by weighting the reliability to adistance of each pixel of the first distance image, and a third distanceimage calculated by estimating a distance of each pixel based on thepolarized image.
 2. The distance measuring device according to claim 1,wherein the reliability acquisition process is configured to set thehigher value as the reliability the smaller the difference between themeasured time and the photographing time is.
 3. The distance measuringdevice according to claim 1, wherein the distance acquisition process isconfigured to estimate a scale of a distance of each pixel of the thirddistance image by performing a process of minimizing the squared errorbetween the second distance image and the third distance image, andcalculate the output value by converting the third distance image usingthe estimated scale.
 4. The distance measuring device according to claim3, wherein the distance acquisition process is configured to calculate avariable at which an error between a distance of each pixel of thesecond distance image, and a value obtained by multiplying a distance ofeach pixel of the third distance image by the variable for estimatingthe scale is minimized, and calculate a value obtained by multiplyingthe distance of each pixel of the third distance image by the variableas the output value.
 5. The distance measuring device according to claim1, wherein the processor is further configured to calculate a normalimage with which normal information is associated for each pixel basedon the polarized image, and calculate the third distance image based onthe normal image.
 6. The distance measuring device according to claim 1,further comprising: the TOF sensor, wherein the TOF sensor emits a lightbeam for distance measurement as an irradiation pattern extending in apredetermined first direction, and receives reflected light of the lightbeam while sequentially moving the irradiation pattern in a seconddirection orthogonal to the first direction to generate the firstdistance image.
 7. A distance measuring method comprising: obtaining afirst distance image acquired from a time of flight (TOF) sensor and apolarized image generated by calculating a degree of polarization foreach pixel based on a plurality of images acquired from a plurality ofcameras that receives linearly polarized light with differentpolarization directions; and executing a reliability acquisition processthat includes calculating reliability according to a difference betweena time of measuring a distance and a time of photographing the pluralityof images for each pixel of the first distance image, and executing adistance acquisition process that includes calculating an output valueof a distance from the TOF sensor to a subject for each pixel by using asecond distance image calculated by weighting the reliability to adistance of each pixel of the first distance image, and a third distanceimage calculated by estimating a distance of each pixel based on thepolarized image.
 8. The distance measuring method according to claim 7,wherein the reliability acquisition process is configured to set thehigher value as the reliability the smaller the difference between themeasured time and the photographing time is.
 9. The distance measuringmethod according to claim 7, wherein the distance acquisition process isconfigured to estimate a scale of a distance of each pixel of the thirddistance image by performing a process of minimizing the squared errorbetween the second distance image and the third distance image, andcalculate the output value by converting the third distance image usingthe estimated scale.
 10. The distance measuring method according toclaim 9, wherein the distance acquisition process is configured tocalculate a variable at which an error between a distance of each pixelof the second distance image, and a value obtained by multiplying adistance of each pixel of the third distance image by the variable forestimating the scale is minimized, and calculate a value obtained bymultiplying the distance of each pixel of the third distance image bythe variable as the output value.
 11. The distance measuring methodaccording to claim 7, wherein the processor is further configured tocalculate a normal image with which normal information is associated foreach pixel based on the polarized image, and calculate the thirddistance image based on the normal image.
 12. The distance measuringmethod according to claim 7, further comprising: by the TOF sensor,emitting a light beam for distance measurement as an irradiation patternextending in a predetermined first direction, and receiving reflectedlight of the light beam while sequentially moving the irradiationpattern in a second direction orthogonal to the first direction togenerate the first distance image.
 13. A non-transitorycomputer-readable storage medium for storing a program which causes aprocessor to perform processing for distance measuring, the processingcomprising: obtaining a first distance image acquired from a time offlight (TOF) sensor and a polarized image generated by calculating adegree of polarization for each pixel based on a plurality of imagesacquired from a plurality of cameras that receives linearly polarizedlight with different polarization directions; and executing areliability acquisition process that includes calculating reliabilityaccording to a difference between a time of measuring a distance and atime of photographing the plurality of images for each pixel of thefirst distance image, and executing a distance acquisition process thatincludes calculating an output value of a distance from the TOF sensorto a subject for each pixel by using a second distance image calculatedby weighting the reliability to a distance of each pixel of the firstdistance image, and a third distance image calculated by estimating adistance of each pixel based on the polarized image.
 14. Thenon-transitory computer-readable storage medium according to claim 13,wherein the reliability acquisition process is configured to set thehigher value as the reliability the smaller the difference between themeasured time and the photographing time is.
 15. The non-transitorycomputer-readable storage medium according to claim 13, wherein thedistance acquisition process is configured to estimate a scale of adistance of each pixel of the third distance image by performing aprocess of minimizing the squared error between the second distanceimage and the third distance image, and calculate the output value byconverting the third distance image using the estimated scale.
 16. Thenon-transitory computer-readable storage medium according to claim 15,wherein the distance acquisition process is configured to calculate avariable at which an error between a distance of each pixel of thesecond distance image, and a value obtained by multiplying a distance ofeach pixel of the third distance image by the variable for estimatingthe scale is minimized, and calculate a value obtained by multiplyingthe distance of each pixel of the third distance image by the variableas the output value.
 17. The non-transitory computer-readable storagemedium according to claim 13, wherein the processor is furtherconfigured to calculate a normal image with which normal information isassociated for each pixel based on the polarized image, and calculatethe third distance image based on the normal image.
 18. Thenon-transitory computer-readable storage medium according to claim 13,further comprising: by the TOF sensor, emitting a light beam fordistance measurement as an irradiation pattern extending in apredetermined first direction, and receiving reflected light of thelight beam while sequentially moving the irradiation pattern in a seconddirection orthogonal to the first direction to generate the firstdistance image.